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* 15 SYSTEM AND METHOD FOR 

INSPECTION USING OFF-ANGLE LIGHTING 

FIELD OF THE INVENTION 



[0001] The present invention pertains to the field of 
component inspection systems. More specifically, the 

invention relates to a system and method for component 
inspection that uses off-angle lighting to illuminate 
features of the component that are of greatest interest. 
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BACKGROUND OF THE INVENTION 
[0002] Inspections systems that use image data are known 
in the art. Such inspection systems typically use image 
data from a component that has been illuminated by either a 
coherent or noncoherent source, and then perform image 
analysis processes on the image data to determine whether 
the component conforms to predetermined criteria. For 
example, image data analysis is used to determine whether 
components have been properly marked, have features in the 
correct location, or have other specified criteria. In this 
regard, a "feature" can include a desired feature, such as a 
contact, or an undesired feature, such as damage on the 
contact that extends from or into the surface of the 
contact . 

15 [0003] One problem with such component inspection systems 
is that three-dimensional aspects of the component must be 
inferred from the image data. Thus, in many cases it is 
difficult to determine whether an indication in the image 
data is from a feature that extends above or below a plane 

20 of reference. Likewise, since such image data analysis uses 
relative changes in brightness to infer the location of 
features and to determine whether such features fall within 
predetermined allowable criteria, it often occurs that the 
features are not distinguishable under a single lighting 

25 source. While it is known to use multiple lighting sources, 
such as lighting sources from two different angles, such 
lighting sources may still result in image data that does 
not clearly identify features of the component that do not 
meet specified criteria, such as to allow the size of 

30 features to be accurately determined. 
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SUMMARY OF THE INVENTION 
[0004] In accordance with the present invention, a system 

and method for component inspection are presented that overcome 
known problems with component inspection. 
5 [0005] In particular, a system and method for component 

inspection are provided that illuminate areas of interest lying 
above and below a planar surface of a feature by using off-axis 
lighting . 

[0006] In accordance with an exemplary embodiment of the 

10 present invention, a system for inspecting components is 
provided. The system includes an axial lighting system that 
illuminates the component with axial lighting to allow one or 
more features of the component to be located, such as by 
causing particles or scratches to be darker than the 

15 background. An off-axis lighting system illuminates the 
component with off-axis lighting in the absence of the axial 
lighting to allow the component to be inspected to locate one 
or more features, such as by causing protruding features and 
damage to bump contacts to be brighter than the background. 

20 [0007] The present invention provides many important 

technical advantages. One important technical advantage of the 
present invention is a system and method for inspecting 
components that uses off-angle lighting to identify damaged 
areas in a manner that allows image data analysis to be 

25 performed to identify protrusions and recesses in surfaces. 
The present invention provides for accurate estimates of the 
size of such protrusions and recesses to be made from a single 
set of image data. 

[0008] Those skilled in the art will further appreciate the 

30 advantages and superior features of the invention together with 
other important aspects thereof on reading the detailed 
description that follows in conjunction with the drawings. 
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BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 
[0009] FIGURE 1 is a diagram of a system for performing 

component inspection in accordance with an exemplary 
embodiment of the present invention; 
5 [0010] FIGURE 2 is an exemplary histogram of pixel 

brightness data in accordance with an exemplary embodiment of 
the present invention; 

[0011] FIGURE 3 is a histogram showing pixel analysis data 

in accordance with an exemplary embodiment of the present 
10 invention; 

[0012] FIGURE 4 is a diagram of system for performing off- 

axis lighting analysis in accordance with an exemplary 
embodiment of the present invention; 

[0013] FIGURE 5 is a flowchart of a method for inspecting a 

15 component using off-axis lighting in accordance with an 
exemplary embodiment of the present invention; 

[0014] FIGURE 6 is a flowchart of a method for analyzing 

off-axis image data in accordance with an exemplary embodiment 
of the present invention; 
20 [0015] FIGURE 7 is a flowchart of a method for determining 

whether a protrusion is present on the surface of a feature in 
accordance with an exemplary embodiment of the present 
invention; 

[0016] FIGURE 8 is a flowchart of a method for analyzing 

25 subsurface damage in a feature in accordance with an exemplary 
embodiment of the present invention; 

[0017] FIGURE 9 is a diagram of a system for analyzing dual 

lighting image data in accordance with an exemplary embodiment 
of the present invention; and 
30 [0018] FIGURE 10 is a flow chart of a method for analyzing 

dual lighting image data in accordance with an exemplary 
embodiment of the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 
[0019] In the description that follows, like parts are 
marked throughout the specification and drawings with the 
same reference numerals, respectively. The drawing figures 
5 are not necessarily to scale, and certain components can be 
shown in generalized or schematic form and identified by 
commercial designations in the interest of clarity and 
conciseness . 

[0020] FIGURE 1 is a diagram of a system 100 for 

10 performing component inspection in accordance with an 
exemplary embodiment of the present invention. System 100 
utilizes distributed off-axis lighting to create image data 
that clearly identifies components, features, and damage to 
such features that extends above or below a reference plane. 

15 [0021] System 100 includes substrate 102, which can be a 
semiconductor die or other suitable materials on which one 
or more features have been placed, machined, deposited, or 
otherwise formed. Substrate 102 includes feature 104. In 
one exemplary embodiment, feature 104 can be a gold bump 

20 that is used to form a high quality electrical contact. In 
this exemplary embodiment, the size of feature 104 is 
significantly smaller than the size of other similar bumped 
contacts that are not formed from gold. For example, 
dimension D2 showing the length of feature 104 can be on "he 

25 order of 50 to 90 micrometers, and H2, representing rhe 
height of the bump, can be on the order of 20 micrometers. 
Such dimensions are significantly smaller than solder bumps 
that are typically used to form electrical contacts with 
semiconductor circuits . 

30 [0022] System 100 further includes off-axis lighting 

5 
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source 106, which can be a suitable off-axis lighting source 
that provides illumination of the entire surface of feature 
104 and substrate 102 that is being inspected. In one 
exemplary embodiment, off-axis lighting source 106 can be a 
circular xenon flash lamp having a diameter Dl of 
approximately 1.5 inches, a thickness Tl of approximately 
5mm, and a height above substrate 102 HI of approximately 
2mm. Xenon lighting may be required when zoom optics are 
used to increase the depth of field, as compared to 
microscope optics which do not require as much light 
intensity but provide less depth of field. Thus, where 
microscope optics provide sufficient depth of field, then an 
incandescent light source or other suitable light source can 
be used. (It should be noted that the dimensions in FIGURE 
15 1 are not shown to scale.) 

[0023] Off-axis lighting source 106 illuminates feature 
104 and substrate 102 in a manner that causes protrusions or 
recesses to be illuminated differently, such that 
protrusions and recesses can be distinguished from each 
20 other, and can be accurately analyzed to determine the size 
of the protrusion or recess. For example, one advantage of 
off-axis lighting source 106 is that the protrusion shown on 
feature 104 will appear brighter than the planar surface of 
feature 104 or the planar surface of substrate 102. 
25 Likewise, if feature 104 included a recess, then the recess 
would appear darker than the planar service of feature 104 
or substrate 102. However, it may be difficult to 
distinguish between the planar surface of feature 104 and 
the planar surface of substrate 102 using off-axis lighting 
30 source 106. Whether or not it will be difficult to 
distinguish between the planar surface of feature 104 and 
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the planar surface of substrate 102 will depend largely on 
the specific dimensions of feature 104, the materials from 
which feature 104 and substrate 102 are made from, and other 
factors. However, in some cases, while illumination with 
5 off-axis lighting source 106 can identify protrusions and 
recesses in planar surfaces, it may not be sufficient to 
provide useful distinction between planar surfaces of 
feature 104 and substrate 102. 

[0024] Axial lighting source 108 provides conventional 
10 axial lighting of feature 104 and substrate 102. This axial 
lighting allows the surface of feature 104 and substrate 102 
to be distinguished more readily, but may not provide 
sufficient indication of the existence of a protrusion or 
recess in the planar surface of feature 104. Thus, axial 
15 lighting source 108 can be used to locate the position of 
feature 104 on substrate 102, but might not be as useful in 
determining the existence of protrusions or recesses in the 
surface of feature 104. 

[0025] Image data system 110 can be implemented in 
20 hardware, software, or a suitable combination of hardware 
and software, and can be one or more software systems 
operating on a general purpose processing platform. As used 
herein, a software system can include one or more objects, 
agents, lines of code, threads, subroutines, databases, 
25 application programming interfaces (APIs), web browser plug- 
ins, or other suitable data structures, source code (human 
readable), object code (machine readable), and can include 
two or more different lines of code or suitable data 
structures operating in two or more separate software 
30 applications, on two or more different processing platforms, 
or in other suitable architectures. In one exemplary 

7 



013377 0087 DALLAS 487768 vl 



Attorney Docket No. 
013377.0087 



PATENT APPLICATION 



embodiment, a software system can include one or more lines 
of code or other suitable software structures operating in a 
general purpose software application, such as an operating 
system, and one or more lines of code or other suitable 
software structures operating in a specific purpose software 
application. In another exemplary embodiment, a software 
system can be implemented as a distributed software system, 
on a different processing platform than that shown in the 
exemplary embodiments herein, or in other suitable manners. 
[0026] Image data system 110 is coupled to axial lighting 
source 108 and off-axis lighting source 106. As used herein, 
the term ^couple," and its cognate terms such as "couples" 
and ^coupled," can include a physical connection (such as 
through a copper conductor) , a virtual connection (such as 
through randomly assigned memory locations of a data memory 
device) , a logical connection (such as through one or more 
logical devices of a semiconducting circuit) , other suitable 
connections, or a suitable combination of such connections. 
In one exemplary embodiment, systems and components are 
coupled to other systems and components through intervening 
systems and components, such as through an operating system 
of a general purpose server platform. 

[0027] Image data system 110 generates image data of 
feature 104 and substrate 102. In one exemplary embodiment, 
image data system 110 can control axial lighting source 108 
and off-axis lighting source 106 so that first image data is 
generated using axial lighting source 108, such as to locate 
features 104 on substrate 102. Then, second image data is 
generated using only off-axis lighting source 106, such as 
to analyze the surface of feature 104 to determine whether 
protrusions or recesses are present. Image data system 110 
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generates image data, such as an N x M array of pixels 
(where N and M are integers) having brightness values 
ranging from 0 (representing the darkest or least bright 
pixel value) to 255 (representing the brightest or most 
5 bright pixel value) , or other suitable ranges of brightness 
values . 

[0028] Image analysis system 112 can be implemented in 
hardware, software, or a suitable combination of hardware 
and software, and can be one or more software systems 
y= 10 operating on a general purpose processor platform. Image 
O analysis system 112 is coupled to image data system 110, and 

g obtains the image data generated by image data system 110, 

U] both for direct and off-axis illumination. Image analysis 

iUi r. 

y3 system 112 then analyzes the image data to identify the 

~ty 15 location of feature 104 and other similar features, and to 
l- k further analyze feature 104 and other features to determine 

r{ whether protrusions or recesses exist in the surface of such 

y features that renders them unusable. In one exemplary 

rj embodiment, image analysis system 112 can use pixel image 

20 processing to identify the location of feature 104 and other 
features, and whether such features are damaged. This pixel 
image processing can include determining whether adjacent 
pixels have a gradient that exceeds known allowable gradient 
differentials, whether a histogram of pixels of the image 
25 data indicate the existence of damage, such as protrusions 
or recesses, or other suitable processes. Image analysis 
system 112 analyzes image data from off-axis lighting source 
106, so as to provide identification of damage that may 
otherwise be difficult to identify. 
30 [0029] In operation, system 100 is used to inspect 
components to determine whether the components meet 
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predetermined allowable criteria. In one exemplary 

embodiment, system 100 can be used to inspect gold bumps or 
other small features that have heretofore been difficult or 
impossible to inspect using image data analysis. System 100 
allows the entire surface of a component to be illuminated 
in a manner that allows bulk processing of the image data of 
the component to be performed, such as histogram processing, 
gradient processing, or other suitable bulk imaging 
techniques. In this manner, system 100 reduces the need for 
performing laser tracking or point analysis of images, such 
as where the laser must be traced or placed at predetermined 
points that cover all features of interest of the component. 
[0030] System 100 further improves on inspection 
techniques by providing two lighting sources for two 
different component inspection functions. System 100 uses a 
first lighting source to align the component for inspection, 
and a second lighting source to inspect the component to 
identify damage to the component. In this manner, system 
100 uses two or more lightning sources in a manner where 
each lighting source performs a different function. 
[0031] FIGURE 2 is an exemplary histogram 200 of pixel 
brightness data in accordance with an exemplary embodiment 
of the present invention. Histogram 200 shows a plot of 
brightness magnitude versus the number of pixels having that 
brightness magnitude for an exemplary image. Brightness 
value increases with increasing x-axis values, and number of 
pixels increases with the increasing y-axis values. In this 
exemplary embodiment, histogram 200 includes two maxima. 
The first maximum, having the lower brightness values, is 
representative of the background pixel brightness values. 
The second maximum, having the higher brightness values, is 

10 
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representative of the feature pixel brightness data. The 
brightness data of histogram 200 can be generated from off- 
axis lighting source 106, or axial lighting, such as from 
axial lighting source 108 (in which case the histogram is 
5 reversed, such that the first maximum represents the bump 
and the second maximum represents the background. Thus, 
histogram 200 can be used to locate the position of 
features, by using the coordinate data from the pixels of 
the second maximum. In this manner, an initial location of 
_ ; 10 features can be performed. 

D [0032] Likewise, a gradient selection method can also be 

g used to identify features and the edge of such features. 

U] For example, the CVLMIL software available from 

J Semiconductor Technologies and Instruments of Piano, Texas 

fy 15 includes a gradient analysis tool that analyzes image data 
^ to locate edges, such as by detecting gradient changes in 

D pixels of image data. In this exemplary embodiment, if the 

yd ar ea within a feature is brighter than the surrounding area, 

then the edge can be located by analyzing a predetermined R 
20 x S block of pixels (where R and S are integers) along 
either the R or S axis until a gradient is detected between 
pixel values within the R x S block that is greater than a 
predetermined allowable gradient. If the pixels outside of 
the feature are dark and the pixels within the feature are 
25 light, then as the R x S block of pixels moves onto the 
feature, the gradient of the pixels within the block will go 
from a predetermined allowable gradient to a gradient that 
exceeds the predetermined allowable amount and back to the 
predetermined allowable gradient. Thus, the approximate 
30 location of the edge can be inferred from the weighted 
midpoint of the transition between the flat gradient in the 

11 
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dark area and the flat gradient in the light area. Other 
suitable procedures can be used to locate the edge of a bump 
or feature. 

[0033] FIGURE 3 is a histogram 300 showing pixel analysis 
5 data in accordance with an exemplary embodiment of the 
present invention. Histogram 300 shows 3 relative maxima in 
the histogram data. The first relative maximum, having a 
peak at the highest brightness value, is indicative of 
protrusions or other above surface damaged areas when they 

10 are illuminated with a light source such as off-axis 
lighting source 106. In this exemplary embodiment, the off- 
axis light source causes such protrusions to be brighter 
than the feature from which they protrude, such as feature 
104. Histogram 300 further includes a second relative 

15 maximum, which is indicative of all the clean portions of 
the planar surface. The third maximum at the darkest end of 
the brightness axis is indicative of damage that extends 
below the surface or is recessed, such as scratches, 
indentations, probe marks or other similar damage. When 

20 illuminated by a light source such as off-axis lighting 
source 106, then such recesses into the surface will 
generate lower than average pixel brightness. 

[0034] Histogram 300 is used to locate areas requiring 
additional inspection by associating the pixels in the above 

25 surface and below surface relative maxima with their 
locations in the N x M array of pixel data. These pixels 
can then be further analyzed to determine whether they are 
indicative of damage, scratches, dust spots, or for other 
suitable purposes. 

30 [0035] FIGURE 4 is a diagram of system 400 for performing 
off-axis lighting analysis in accordance with an exemplary 

12 
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embodiment of the present invention. System 400 includes 
feature locator system 404, defect locator system 406, 
protrusion analysis system 408, recess analysis system 4L0, 
and missing feature system 412, each of which can be 
5 implemented in hardware, software, or a suitable combination 
of hardware and software, and which can be one more software 
systems operating on a general purpose processing platform. 
[0036] Feature locator system 404 identifies the location 
of one or more features of a component, such as gold bump 
10 contacts. In one exemplary embodiment, feature locator 
system 404 identifies features when axial lighting is used, 
and then generates a suitable edge boundary indicator for 
use in analyzing the image data when off-axis lighting is 
used. Likewise, feature locator system 404 can also be used 
15 to locate the edges of features using only off-axis lighting 
when the design and material composition of the component 
allow such functionality. Feature locator system 404 can 
also perform histogram analysis of data to identify the 
location of one or more features, such as by plotting the 
20 histogram of pixel brightness values as a function of the 
number of pixels to locate features, and then performing 
additional image analysis to verify that the feature 
indication is actually a feature, such as by using gradient 
analysis to identify the edges of the feature. 
25 [0037] Defect locator system 406 is used to identify one 

or more defects in image data generated by off-axis 
lighting. In one exemplary embodiment, defect locator 
system 406 receives image data after analysis by feature 
locator system 404, and performs additional histogram data 
analysis on one or more features that have been identified. 
In this exemplary embodiment, features of a component such 

13 
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as gold bumps can be first identified using histogram data, 
and then a second set of histogram data can be generated for 
individual features, so as to identify damage to those 
features, where each damage location itself is a "feature" 
5 for the purpose of image data analysis. Defect locator 
system 406 can include predetermined tolerances for 
histogram data that have been empirically matched with 
protrusion damage, such as nodules, recess damage such as 
scratches or mouse bites, or other damage. 

10 [0038] Protrusion analysis system 408 receives image data 
and generates protrusion data. In one exemplary embodiment, 
the image data can include predetermined areas that have a 
brightness value that exceeds the expected brightness value 
of the surface of the feature, such that protrusion analysis 

15 system 408 can then analyze the feature to determine whether 
it is of a size that exceeds predetermined allowable 
protrusions. In this exemplary embodiment, a morphologic 
opening algorithm can be used to remove a predetermined 
number of perimeters from a suspected protrusion feature, 

20 such that if all of the pixels defining the protrusion have 
been removed prior to completion of the morphological 
opening algorithm, then the size of the corresponding 
anomaly can be determined to be below a specified allowable 
tolerance. Other suitable processes can be used. 

25 [0039] Recess analysis system 410 is used to analyze 
image data generated by off-axis lighting to identify one or 
more recess features, such as scratches, mouse bites, probe 
marks or other such recess damage. In one exemplary 
embodiment, recess analysis system 410 identifies areas 

30 where the pixel brightness is lower than a predetermined 
tolerance, such as by using a histogram analysis. These 

14 
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areas are then analyzed to further determine the size and 
gradient of the area, so as to determine whether it is 
representative of damage. For example, the gradient can be 
determined by analyzing the change in brightness over the 
5 length of the suspected damaged area, such as to be compared 
to empirically determined allowable damage (e.g. allowable 
lengths of scratches, allowable width of scratches), or other 
suitable data. Likewise, the total approximated area 

encompassed by a scratch or other recess feature can be 
10 determined. Recess analysis system 410 can then compare the 
recess analysis data to predetermined allowable sizes, and can 
generate recess data indicating the existence of a recess that 
is unacceptable. 

[0040] In one exemplary embodiment, a scratch or other 

15 recess on a feature can be identified by performing a row and 
column projection, such as by summing all pixels in a row of 
each column of an S x T array of pixels (where S and T are 
integers), and taking the average of those pixel values. 
Likewise, a similar procedure can be used for all pixels in a 
20 column of each row of the S x T array. For example, in the 4 
x 6 array of pixel values represented as follows: 
100 100 100 100 100 100 
100 80 60 80 100 100 
100 20 0 20 60 100 
25 100 100 100 100 100 100 

The row and column projections described above would yield: 
100 75 65 75 90 100 

and 

100 

30 86.7 
50 
100 
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[0041] It may be empirically determined that row or 
column projections resulting in gradients that exceed a 
predetermined number of adjacent pixels (such as four or 
more) or that have a predetermined maximum or percentage 
change in value (such as more than 50 relative brightness 
points on a 256 point scale) can result in unacceptable 
levels of misoperation, such that the features on which such 
damage is present should be rejected. Thus, in this 
exemplary embodiment, the damage to the feature would be 
held to be unacceptable based on the failure of either of 
two criteria. Likewise, if both criteria included 

boundaries of allowable values for row and column 
projections (such as reject for 6 or more pixels, borderline 
5 to 4 pixels, and accept 3 or less pixels), then the 
15 feature could be classified as unacceptable based upon the 
fact that both criteria are borderline, based on the area of 
the damage, or based on other or additional damage 
indicators. The values given here are by way of example 
only, and other suitable types of processes can be used also 
20 or alternatively to those described herein. 

[0042] Missing feature system 412 can receive image data 
and determine whether one or more features are missing from 
the image data. In one exemplary embodiment, after a 
feature template has been generated either by operator 
25 selection or image analysis with feature locator system 404, 
a template can be generated that outlines the expected 
location of one or more features. Missing feature system 
412 can analyze image data, such as pixels of brightness 
data, of the expected feature locations, and can generate 
missing feature data if the number of pixels having a 
certain brightness value or greater in an area where a 

16 
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feature is expected is significantly less than the expected 
number. In this manner, missing feature system 412 can 
generate missing feature data that indicates an unacceptable 
device or other suitable data. 
5 [0043] In operation, system 400 receives image data 
generated with off-axis lighting and analyzes the image data 
to determine whether a protrusion, recess or other similar 
damage exists in the image data. System 400 thus allows 
image data features to be analyzed to determine whether the 

10 features meet predetermined criteria. 

[0044] FIGURE 5 is a flowchart of a method 500 for 
inspecting a component using off-axis lighting in accordance 
with an exemplary embodiment of the present invention. 
Method 500 begins at 502 where axial image data is obtained. 

15 The axial image data can include image data obtained from 
axial lighting of a component or other suitable axial image 
data. The method then proceeds to 504. 

[0045] AT 504, feature window data is generated. In one 
exemplary embodiment, the feature window data can include a 

20 perimeter line generated based on user selected criteria, 
pixel gradient data, or other suitable data that is used to 
identify the outline of a feature and to generate a window. 
The window can also be enlarged by a predetermined amount. 
The method then proceeds to 506. 

25 [0046] At 506, off-axis image data is generated. The 
off-axis image data is generated by illuminating the 
component with off-axis lighting, and generating image data 
when the component is so illuminated. For example, the 
component can be illuminated using a circular xenon flash 

30 lamp that is placed parallel and adjacent to the surface of 
the component. Likewise, other suitable processes can be 

17 
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used to generate the off-axis image data by using lighting 
that uniformly illuminates the surface of the feature so as 
to cause recessed areas to be darker than planar surfaces 
and to cause protrusions to be brighter than planar 
5 surfaces. The method then proceeds to 508. 

[0047] At 508, the edges of the feature are located. In 
one exemplary embodiment, the edges can be located using a 
gradient method or process, such as by using the Matrox 
Imaging Library "Blob Analysis" tool or other suitable image 
10 analysis procedures. The method then proceeds to 510. 

0 [0048] At 510, it is determined whether the feature is 
acceptable. For example, if it is determined that the 

01 feature is not acceptable, such as if the feature has 

Hi; unacceptable dimensions, the feature is missing, or other 

■-lit? 

fU 15 unacceptable conditions exist, the method proceeds to 512 
* where the feature is rejected. An error message can be also 

O or alternatively be generated, log data can be generated 

indicating that the feature is unacceptable, or other 
O suitable rejection procedures can be used. The method then 

r " 20 proceeds to 514. At 514, the next feature or die is 
analyzed, such as by initiating method 500 again. 
[0049] If it is determined at 510 that the edges of the 
feature are acceptable, the method proceeds to 516, where a 
histogram analysis is performed. In one exemplary 

25 embodiment, the histogram analysis can be performed for each 
feature, can be performed on all features of the component, 
or other suitable histogram analyses can be performed. The 
method then proceeds to 518. 

[0050] At 518 it is determined whether defects or N> damage 
30 features'' have been located within the boundary that defines 
the feature. In one exemplary embodiment, one or more 
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tolerance levels can be set, either through analysis, 
empirically or otherwise, that identify acceptable levels 
for histogram brightness distribution data. If the pixel 
brightness data of the histogram exceeds these predetermined 
5 levels, it can be determined that a defect is present or may 
be present. If a defect indication has not been located, 
the method proceeds to 520 and the next feature or die is 
analyzed, such as by beginning method 500 over from the 
beginning. Otherwise, the method proceeds to 522. 
10 [0051] At 522, the defect indication is analyzed. For 
q example, the defect can include an area where the pixels are 

O brighter than allowable, indicating a protrusion or other 

[J similar above surface defect, an area where the pixels are 

tn darker than expected, indicating a recess or area where 

XXL 

r: 15 other subsurface damage has occurred, or other similar 
s defects- The defects can be analyzed to determine the size, 

p the area, the brightness gradients, or other suitable 

fU dimensions of the defects. The method then proceeds to 52 4. 

g [0052] At 524 it is determined whether the defects are 

H 20 acceptable. In one exemplary embodiment, the results from 
the defect indication analysis are compared to acceptable 
boundaries or limits for such dimensions. If it is 
determined that the defects are acceptable, the method 
proceeds to 526 and the next feature or die is analyzed, 
25 such as by initiating method 500 for that feature or die. 
Otherwise, the method proceeds to 528 and the feature is 
rejected, such as by generating feature rejection data, 
marking the feature as rejected, or performing other 
suitable processes. The method then proceeds to 530 and the 
30 next feature or die is analyzed. 
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[0053] In operation, method 500 is used to analyze 
feature data generated from off-axis lighting. Method 500 
allows protrusions and recesses in feature surfaces to be 
identified by analysis of off-axis lighting data. Likewise, 
5 method 500 allows axial lighting to be used to identify the 
feature, generate - feature edge windows, or for other 
suitable purposes. 

[0054] FIGURE 6 is a flowchart of a method 600 for 
analyzing off-axis image data in accordance with an 
10 exemplary embodiment of the present invention. Method 600 
allows off-axis image data of the component to be analyzed 
to locate protrusions, recesses or other damage features of 
the component . 

[0055] Method 600 begins at 602 where histogram data is 
15 generated using pixels and a feature window. The feature 
window can be generated by circumscribing the feature with a 
boundary, such that all pixels within the boundary are 
analyzed. The method then proceeds to 604. 

[0056] At 604 it is determined whether the number of 
20 pixels having a brightness that exceeds a first tolerance 
level exists. For example, the number of allowable pixels 
having a brightness that exceeds the first tolerance level 
can be empirically determined, the slope of the histogram 
data can be analyzed to determine whether a maximum occurs 
25 having a peak greater than a predetermined tolerance, or 
other suitable procedures can be used. If it is determined 
at 604 that the number of pixels is greater than the first 
tolerance level, the method proceeds to 606, where an 
analysis is performed for damage above the surface of the 
30 feature. For example, the pixels having brightness values 
that exceed the predetermined tolerance can be analyzed to 
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determine whether they form an area having a size that is 
greater than a predetermined allowable tolerance. Other 
suitable procedures can be used. The method then proceeds 
to 608 . Likewise, if it is determined at 604 that the 
number of pixels exceeding the tolerance level is not 
greater than a predetermined allowable number, the method 
proceeds directly to 608. 

[0057] At 608 it is determined whether a predetermined 
number of pixels having a brightness value between the first 
tolerance level and a second tolerance level exists. If the 
number of pixels exceeding the second tolerance level but 
less than the first tolerance level is less than a 
predetermined number, the method proceeds to 610 where a 
missing feature error message is generated. In one 

exemplary embodiment, if the feature window is generated 
automatically, such as by using a reference feature to 
locate a feature window template on a component, it is 
possible that a missing feature would be included in the 
window, with the result that the window would contain a 
large number of pixels not having a brightness value that is 
expected. In this exemplary embodiment, the missing feature 
error message can be generated, the feature can be flagged 
as being missing, an operator notification can be generated 
or other suitable procedures can be implemented. The method 
then proceeds to 614. Likewise, if it is determined at 608 
that the number of pixels having brightness values between 
the first and second tolerance levels exists, then the 
method proceeds to 612 where the feature is indicated to be 
acceptable, meaning that it is in the correct location and 
is not missing. The method then proceeds to 614. 
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E0058] At 614 it is determined whether a number of pixels 
is less than a third tolerance level, which can be equal to 
or less than the second tolerance level. If the number of 
pixels is less than this level, the method proceeds to 616 
5 where an analysis is performed to identify below surface 
damage. In one exemplary embodiment, the coordinates of the 
pixels that fall below the third tolerance level can be 
identified and the size, gradient and other characteristics 
of the areas encompassed by such pixels can be analyzed to 

10 determine whether the characteristics indicate that damage 
exists. Otherwise, if it is determined at 614 that the 
number of pixels having brightness values less than the 
third tolerance level does not exceed predetermined levels, 
then the method proceeds to 618 and terminates. 

15 [0059] In operation, method 600 allows off-axis lighting 
to be used to generate image data that is then analyzed to 
determine whether a feature has above surface damage, below 
surface damage, whether the feature is missing, or whether 
other damage conditions exist. Method 600 thus allows 

20 features to be analyzed to reliably and quickly locate 
damage that may otherwise be difficult to locate when using 
only axial lighting or other existing image analysis and 
inspection techniques. 

[0060] FIGURE 7 is a flowchart of a method 700 for 
25 determining whether a protrusion is present on the surface 
of a feature in accordance with an exemplary embodiment of 
the present invention. Method 700 begins at 702 where 
defects are located using pixels having a brightness value 
that is greater than a predetermined tolerance, such as from 
30 image data generated by off-axis lighting. The method then 
proceeds to 704 where a first pixel in a test area is 
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selected. In one exemplary embodiment, the pixel can be 
selected by locating a pixel on the edge of an area, such as 
by using a gradient differential technique, by determining 
the mathematical edge using pixel coordinate data, or other 
suitable processes. The method then proceeds to 706. 
[0061] At 706, the pixel in the test area is blacked out. 
For example, the pixel brightness data can be set to 0, a 
set of pixels in the test area can be identified and data 
corresponding to the selected pixel can be stored to 
indicate that the pixel has been checked, or other suitable 
procedures can be used. The method then proceeds to 708. 
[0062] At 708 it is determined whether a predetermined 
integer or other suitable number of times of dilation around 
the test area have been performed. For example, if the test 
area is a circular region having a diameter of 10 pixels, 
then performing an erosion around the test area one time 
would result in an area having a pixel diameter of 8 pixels, 
performing a second dilation two times around would result 
in an area having a diameter of 6 pixels, and so forth. The 
number of times around the test area can be determined by 
tracking the progress of erosion relative to the first 
pixel, or by other suitable methods. If it is determined 
that the pixels have been blacked out N times around, the 
method proceeds to 716. Otherwise, the method proceeds to 
710 where an adjacent pixel is selected in the test area. 
In one exemplary embodiment, each of the pixels adjacent to 
the pixel just blacked out can be checked to determine which 
pixel is brightest, which pixel is on an edge of the 
remaining test area pixels, whether any of the pixels have a 
brightness value that exceeds a predetermined brightness 
value, or other suitable procedures can be used. The method 
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then proceeds to 712. 

[0063] At 712 is determined whether an adjacent pixel was 
located. If an adjacent pixel is not located, the method 
proceeds to 714 and the indication is cleared as being 
damage having a greater than an allowable size. Otherwise, 
the method returns to 706. 

[0064] After a predetermined number of erosions have been 
performed on the test area, the method proceeds to 716 where 
it is determined whether there are any pixels left. For 
example, if the dilations have resulted in the elimination 
= of all pixels in the test area, the method proceeds to 720 

□ where the indication of a potential anomaly is cleared. 

% Otherwise the method proceeds to 718 where an indication is 

n generated of the presence of a potential anomaly, nodule, or 

frj 15 other protrusion type of damage. 

[0065] In operation, method 700 performs a morphologic 
opening analysis to determine the size of a protrusion, such 
V as by using pixel brightness generated from off-axis 

5 lighting. Method 700 thus allows a nodule or other similar 

20 protrusions to be identified. The number of times around at 
step 708 can be set to equal a customer specified allowable 
size for a nodule, can be set based on analytical or 
empirical data, or other suitable data. 

[0066] FIGURE 8 is a flowchart of a method 800 for 
25 analyzing subsurface damage in a feature in accordance with 
an exemplary embodiment of the present invention. Method 
800 begins at 802 where the row and column bounds for a 
suspect area are determined. In one exemplary embodiment, 
the suspect area can be determined by generating histogram 
30 data from off-axis lighting, and selecting pixels having a 
brightness less than a tolerance level. The coordinates of 
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such pixels can then be used to identify the test area. The 
row and column bounds for the test area can be generated by 
selecting the row and column of pixels outside of the test 
area, such as by using a gradient method. The method then 
5 proceeds to 804. 

[0067] At 804, a row projection of pixel brightness 
values is obtained. In one exemplary embodiment, the pixel 
brightness values for all pixels in the columns of each row 
entry can be averaged, such as to perform a row projection 
10 of the pixel brightness values. The method then proceeds to 
806, 

[0068] At 806, a column projection of pixel brightness 
values is obtained. In one exemplary embodiment, the pixel 
brightness values of each pixel in a row extending from a 
15 column can be averaged for that column, such as to create a 
column projection for pixel brightness values. The method 
then proceeds to 808. 

[0069] At 808, it is determined whether the size of the 
recess damage area is greater than a predetermined tolerance 

20 value. In one exemplary embodiment, size can be estimated 
by using the number of pixels over which the row projection 
and column projection extend, to estimate a size of the 
recess damage. If the size is greater than the tolerance, 
the method proceeds to 810 and the indication is rejected as 

25 having a size greater than tolerance. Otherwise, the method 
proceeds to 812. 

[0070] At 812 it is determined whether the row gradient 
exceeds an allowable tolerance. The allowable tolerance can 
be generated empirically or through analysis, in accordance 
30 with customer specifications, or in other suitable manners. 
For example, the allowable tolerance can include a maximum 
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brightness differential between the brightest and the 
darkest pixels, a maximum allowable differential between any 
two adjacent pixels, or other suitable gradient tolerances. 
If the row gradient exceeds the tolerance, the method 
5 proceeds to 814 and the indication is rejected. Otherwise, 
the method then proceeds to 816. 

[0071] At 816 it is determined whether the column 
gradient exceeds an allowable tolerance. The allowable 
tolerance can be generated empirically or through analysis, 
10 in accordance with customer specifications, or in other 
M* suitable manners. If it is determined at 816 that the 

column gradient exceeds the tolerance, the method proceeds 
yj to 818 and the indication is rejected. Otherwise, the 

j*j method proceeds to 820. 

15 [0072] At 820 it is determined whether the combination of 
' a ~ size, row gradient, and column gradient is acceptable. In 

M one exemplary embodiment, the customer can specify maximum 

py values for each measurement, maximum values for the 

W combination of values can be determined empirically, or 

D 

y, 20 other suitable processes can be used. Further, borderline 
values can be identified where the combination of values is 
analyzed when one or more values are borderline. If it is 
determined at 820 that the combination is acceptable, the 
method proceeds to 822 and the indication is rejected. 
25 Otherwise, the method proceeds to 824 and the indication is 
cleared. 

[0073] In operation, method 800 allows subsurface damage 
to be analyzed to determine whether it is allowable. In one 
exemplary embodiment, method 800 uses off-axis lighting to 
30 generate pixel data which is analyzed to determine whether 
recesses, scratches, mouse bites, probe marks, or other 
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damage has resulted in an unallowable feature. 
[0074] FIGURE 9 is a diagram of a system 900 for 
analyzing dual lighting image data in accordance with an 
exemplary embodiment of the present invention. System 900 
5 can be used to locate sunken ball contacts and other 
anomalies . 

[0075] System 900 includes dual lighting control system 
902, band analysis system 904, edge location system 906 and 
feature analysis system 908, each of which can be 
10 implemented in hardware, software, or a suitable combination 
of hardware and software, and which can be one or more 
p software systems operating on a general purpose processing 

H: platform. Dual lighting control system 902 controls two or 

yHj more light sources, such as an axial light source and an 

jjj 15 off-axis light source, so as to illuminate a component with 
fy the two or more light sources. In one exemplary embodiment, 

!\ the off-axis light source can be a ring of light-emitting 

Q diodes or other suitable light sources. In another 

^ exemplary embodiment, dual lighting control system 902 can 

D 20 control two or more light sources in a predetermined 
sequence, such as to generate two or more different sets of 
image data when the component is illuminated with two or 
more different sets of light sources. 

[0076] Band analysis system 904 receives edge data and 
25 determines whether a band is present in a set of image data. 
In one exemplary embodiment, band analysis system 90 4 can 
receive edge coordinate data that identifies two or more 
edges between dark areas and light areas in a set of image 
data, and can generate band data that identifies whether the 
30 edges define a circular band, such as a band created by a 
sunken ball contact when it is illuminated with an axial 
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light source and an off-axis light source at the same time, 
or other suitable bands. Such bands can be indicative of an 
anomaly, such as a sunken ball contact, that may render the 
component inoperative. Band analysis system 904 can further 
5 receive edge data from two or more different locations and 
other feature data, such as component center data, and can 
determine whether a band exists based on the edge data and 
the other feature data. 

[0077] Edge location system 906 analyzes image data and 
10 generates edge data. In one exemplary embodiment, edge 
location system 906 searches for edges by analyzing 
jl' brightness data differences in adjacent pixels, where an 

D edge is defined by a predetermined brightness data 

^ difference, a set of predetermined brightness data 

HI 15 differences (such as in two or more sets of adjacent 

yp 

~l pixels), or other suitable data. In another exemplary 

2 embodiment, edge location system 906 can analyze pixel 

p brightness data in predetermined locations, relative to 

PJ predetermined component features, using histogram analysis, 

20 or using other suitable processes. 
^ [0078] Feature analysis system 908 receives image data, 

band data, edge data, or other suitable data and determines 
whether a feature is present, such as a sunken ball contact. 
In one exemplary embodiment, feature analysis system 908 can 
25 determine whether a band width exceeds a predetermined 
allowable band width, whether a band extends for a 
predetermined length or periphery, or other band-related 
feature data, such as to indicate whether a sunken ball 
contact or other anomaly is present. Likewise, missing 
30 features, protrusions, scratches, or other damage can be 
detected by feature analysis system 908, and notification 
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data can be generated to alert an operator, remove the 
component from production, or perform other suitable 
procedures. Feature analysis system 908 generates feature 
or component acceptance data, feature or component rejection 
5 data, operator data alerting the operator of the need for 
operator inspection, or other suitable data. 

[0079] In operation, system 900 allows a component to be 
inspected using two or more light sources that illuminate 
the component simultaneously, such as an axial light source 

10 and an off-axis light source. System 900 can be used to 
locate sunken ball contacts or other anomalies by locating 
image data that indicates the presence of such anomalies, 
such as a circular ring that appears around a sunken ball 
contact when it is illuminated by an axial light source and 

15 an off-axis light source. 

[0080] FIGURE 10 is a flow chart of a method 1000 for 
analyzing dual lighting image data in accordance with an 
exemplary embodiment of the present invention. Method 1000 
begins at 1002, where a component is illuminated with an 

20 axial light source and an off-axis light source. In one 
exemplary embodiment, the off -axis light source can be a 
ring arrangement of light-emitting diodes, or other suitable 
circular light sources. The method then proceeds to 1004. 
[0081] At 1004, image data of the component is generated, 

25 such as by generating an N x M array of pixels of brightness 
data. The method then proceeds to 1006, where it is 
determined whether a band is present. In one exemplary 
embodiment, the band can be detected by locating two or more 
edges in the N x M array of pixel data, such as edges at two 

30 or more points, a continuous edge that extends in a line or 
circle, or other suitable data. If no band is present, the 
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method proceeds to 1010. Otherwise, the method proceeds to 
1008. 

[0082] At 1008, one or more band width metrics are 
determined. In one exemplary embodiment, the band width 
5 metric can include the thickness of the band, the 
circumferential extent of the band, the location of the band 
relative to other features of the component, or other 
suitable metrics. The method then proceeds to 1010. 
[0083] At 1010, it is determined whether one or more 

10 other anomalies are present, such as by analyzing histogram 
data, binarizing image data, mapping image data, or other 
suitable processes. If it is determined that no other 
anomalies are present, the method proceeds to 1014. 
Otherwise, the method proceeds to 1012 where other metric 

15 data is determined, such as the size of an anomaly, the 
extent of a histogram data variation, or other suitable 
data. The method then proceeds to 1014. 

[0084] At 1014, it is determined whether any metrics 
exceed predetermined allowable limits, such as whether the 

20 thickness or circumferential extent of a ring exceeds a 
predetermined allowable thickness or circumferential extent. 
If it is determined that no metric exceeds a limit, the 
method proceeds to 1016, where the component is accepted or 
the next feature is analyzed. Otherwise, the method 

25 proceeds to 1018 where the component is rejected or the next 
feature is analyzed. A feature database can likewise be 
updated to reflect the rejection of the feature, such as 
where a predetermined number of features can be rejected 
without rejecting the component. 

30 [0085] In operation, method 1000 allows a component to be 
inspected using component illumination from two or more 
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light sources, such as an axial light source and an off-axis 
light source. Method 1000 analyzes image data to locate one 
or more bands or other anomalies, and then compares the 
located bands or anomalies to one or more metrics (such as 
empirically or analytically determined metrics) to determine 
whether the bands or anomalies exceed the metric. 
[0086] Although exemplary embodiments of a system and 
method of the present invention been described in detail 
herein, those skilled in the art will also recognize that 
various substitutions and modifications can be made to the 
systems and methods without departing from the scope and 
spirit of the appended claims. 
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